Automated Image Rectification in Video-Endoscopy

Video-Endoscopy has proven to be significantly less invasive to the patient. However, it also creates a more complex and difficult operating environment that requires the surgeon to operate through a video interface. Visual feedback control and image interpretation in this operating environment can be troublesome. Automated image analysis has tremendous potential in improving the surgeon's visual feedback, resulting in better patient safety, reduced operation time, and savings in health care. In this paper, we present our design of an image rectification algorithm for maintaining the head-up display in video-endoscopy and report some preliminary results. Video-endoscopy [1, 2] are minimally invasive surgical procedures where small incisions are made on the patient to accommodate surgical instruments such as scalpels, scissors, staple guns, and a video endoscope. The scope acquires video images of the bodily cavity that are displayed on a monitor to provide visual feedback to the surgeon. Though video-endoscopy has proven to be tremendously beneficial in shortening the recuperation time and lowering the treatment cost [1, 2], this patient-oriented technology has increased the difficulty of performing the procedures for the surgeon. E.g. in open surgery, though the surgeon may move about the operating table and view the body anatomy from different viewpoints, the surgeon's sense of up and down is maintained. Or in the computer graphics terminology, the “head-up” vector, which indicates one's general perception of the “up” direction in the environment, is unchanged. But in video-endoscopy, the bodily cavity is not exposed and the surgeon's perception of the “up” direction is established through the understanding of the anatomy. A large, about-the-axis rotation of the scope and the camera will change the orientation of the body anatomy, which often times results in the loss of one's orientation and bearing. Hence, how to compensate for the about-the-axis rotation of the video scope to maintain the right “head-up” display (i.e., a display where the image is rectified and re-oriented in such a way that the view corresponds to that taken with an upright camera) is important. We propose a new formulation for rotation sensing and compensation based on novel image analysis techniques. The image analysis tasks proposed here can be stated succinctly as follows: Given a video sequence taken during an endoscopy surgery, with large panning and rotation of the scope, deduce the amount of the about-the-axis rotation relative to a specified reference frame. The images are then rectified and displayed in such a way as if the camera is held in an upright orientation. The essence of our rotation estimation and compensation scheme is to track anatomical features in an endoscopic video sequence and use the tracked features over time to deduce the camera motion parameters. The recovered motion parameters then allow rotation of the camera's head-up vector relative to that of a reference frame computed. Images can then be re-rendered by compensating for the rotation of the head-up vector. Fig. 1 shows some image segmentation and tracking results. This preliminary result shows our ability to distinguish instrument regions from organ regions (see caption for explanation). Fig. 2 shows 2D and 3D tracking results over a sequence of about 200 frames. Fig. 2 (a) shows typical 2D feature location errors as a function of time, while Fig. 2 (b) shows the absolute error in estimated angle of rotation in 3D camera motion. Both results are quite good. Fig. 1. Upper row: (a) Original image. (b) Binary image from color classification. Black and white pixels represent surgical instruments and organs, respectively. (c) Directional median filters were used to suppress noise. (d) Labeled image, different colors represent isolated instrument regions. (e) Computed bounding boxes that delineate instrument regions in images. Lower row: Tracking of instrument region. 0 20 40 60 80 100 120 140 160 180 100 150 200 250 300 350 400 450 500 both plots lie on top of each other number of frames processed pi xe l p os iti on 2−D tracking: real position and tracked position vs frame # 10 20 30 40 50 60 70 80 2 4 6 8 10 12 14 16 18 20 22 deviation of 3−D reconstruction from true camera position number of frames processed % d ev ia tio n Fig. 2. (a) shows the 2D tracking error positions as function of time over about 200 frames. (b) shows the 3D tracking error (deviation in estimated angle of rotation from ground truth). Acknowledgment: This research was supported in part by Karl Storz Imaging, Inc. and the State of California Micro Program.